Audiovisual integration of emotional signals from others' social interactions

Audiovisual perception of emotions has been typically examined using displays of a solitary character (e.g., the face-voice and/or body-sound of one actor). However, in real life humans often face more complex multisensory social situations, involving more than one person. Here we ask if the audiovisual facilitation in emotion recognition previously found in simpler social situations extends to more complex and ecological situations. Stimuli consisting of the biological motion and voice of two interacting agents were used in two experiments. In Experiment 1, participants were presented with visual, auditory, auditory filtered/noisy, and audiovisual congruent and incongruent clips. We asked participants to judge whether the two agents were interacting happily or angrily. In Experiment 2, another group of participants repeated the same task, as in Experiment 1, while trying to ignore either the visual or the auditory information. The findings from both experiments indicate that when the reliability of the auditory cue was decreased participants weighted more the visual cue in their emotional judgments. This in turn translated in increased emotion recognition accuracy for the multisensory condition. Our findings thus point to a common mechanism of multisensory integration of emotional signals irrespective of social stimulus complexity

Regulating wellbeing in the brave new quantified workplace

Purpose: The purpose of the paper is to lay out the conceptual issues arising alongside the rise of sensory technologies in workplaces designed to improve wellness and productivity. Design/methodology/approach: This is a text based conceptual paper. Our approach is to throw light on some of the emerging issues with the introduction of wearable self-tracking technologies in workplaces. Findings: The paper indicates that scholars will need to put ethical issues at the heart of research on sensory tracking technologies in workplaces that aim to regulate employee behaviour via wellness initiatives. Originality/value: This is an original article. Since there is very little scholarly research in this area, it is important to begin to consider the implications of sensory technology in workplaces linked to wellness initiatives, given the probable impact it will have on work design and appraisal systems

Perception of emotion in social interactions from body movement and voice

The central theme of this thesis was to examine different aspects related to the observation and judgement of emotions from the body movement and voice of two actors engaged in social interaction. There were four major goals related to this theme. The first goal was to create a novel stimulus set for the study of emotional social interactions. The second was to validate the created stimulus set by examining emotion perception in ways similar to that done with single actor displays. The third goal was to examine the effect of degrading visual and auditory information on the perception of emotional social interactions. The final goal was focused on the multimodal integration of emotional signals from body movement and voice. Initially, a stimulus set was created that incorporated body movement and dialogue between two actors in brief, natural interactions that were happy, angry or neutral at different levels of intensity. The stimulus set was captured using a Vicon motion and voice capture system and included a group of nine professional and non-professional actors. This resulted in a corpus of 756 dyadic, multimodal, emotional interactions. A series of experiments were conducted presenting participants with visual point-light displays, auditory voice dialogues or combinations of both visual and auditory displays. Observers could accurately identify happy and angry interactions from dyadic displays and voice. The intensity of expressions influenced the accuracy of the emotional identification but only for angry rather than happy displays. After validation of the stimulus set, a subset was selected for further studies. Various methods of auditory and visual distortion were tested separately for each modality to examine the effect of those distortions on recognition of emotions from body movement and voice. Results for dyadic point-light displays followed similar findings from single actor displays that inversion and scrambling decreased the overall accuracy of emotion judgements. An effect of viewpoint was also found, indicating that observation of interaction from a side viewpoint was easier for emotion detection than observation of interaction from an oblique viewpoint. In the case of voice, methods of brown noise and low-pass filtering were shown to degrade emotion identification. However, with both visual and auditory methods of distortion, participants were still able to identify emotions above the level of chance, suggesting high sensitivity to emotional cues in a social context. In the final set of studies, the stimulus set was used in a multimodal context to examine the perception of emotion from movement and voice in dyadic social interactions. It was repeatedly found that voice dominated body movement as a cue to emotions when observing social interactions. Participants were less accurate and slower in emotion discrimination when they were making judgements from body movement only, compared to conditions when movement was combined with dialogue or when dialogue was presented on its own. Even when participants watched emotionally mismatched displays with combined movement and voice, they predominantly oriented their responses towards the voice rather than movement. This auditory dominance persisted even when the reliability of the auditory signal was degraded with brown noise or low-pass filtering, although visual information had some effect on judgements of emotion when it was combined with a degraded auditory signal. These results suggest that when judging emotions from observed social interactions, we rely primarily on vocal cues from conversation rather than visual cues from body movement

A dyadic stimulus set of audiovisual affective displays for the study of multisensory, emotional, social interactions

We describe the creation of the first multisensory stimulus set that consists of dyadic, emotional, point-light interactions combined with voice dialogues. Our set includes 238 unique clips, which present happy, angry and neutral emotional interactions at low, medium and high levels of emotional intensity between nine different actor dyads. The set was evaluated in a between-design experiment, and was found to be suitable for a broad potential application in the cognitive and neuroscientific study of biological motion and voice, perception of social interactions and multisensory integration. We also detail in this paper a number of supplementary materials, comprising AVI movie files for each interaction, along with text files specifying the three dimensional coordinates of each point-light in each frame of the movie, as well as unprocessed AIFF audio files for each dialogue captured. The full set of stimuli is available to download from: http://​motioninsocial.​com/​stimuli_​set/​

Failing to encourage physical activity with wearable technology:what next?

Wearable devices and associated systems that provide real-time feedback aim to encourage healthy behaviours. However, while the research base has grown considerably, results continue to paint a mixed picture when demonstrating wearables’ ability to increase levels of physical activity. Given these recent developments, this commentary explores the key reasons why wearable devices and other mobile technologies often fail to change behaviour. We also provide several suggestions that could feed into future research designs and maximise the success of subsequent interventions. These recommendations aim to stimulate interdisciplinary discussions by encouraging clinicians and researchers to consider how these technological advances can be effectively leveraged, and become a core component of preventative medicine in the 21st century

A simple location-tracking app for psychological research

Location data gathered from a variety of sources is particularly valuable when it comes to understanding individuals and groups. However, much of this work relies on participants’ active engagement to regularly report their location. More recently, smartphones have been used to assist with this process, but while commercial smartphone applications are available, these are often expensive and not designed with researchers in mind. In order to overcome these and other related issues, we have developed a freely available Android application that logs location accurately, stores data securely, and ensures participants can provide consent or withdraw from a study at any time. Further recommendations and R code are provided to assist with subsequent data analysis

Look at those two!:The precuneus role in unattended third-person perspective of social interactions

Human beings often observe other people's social interactions without being a part of them. Whereas the implications of some brain regions (e.g. amygdala) have been extensively examined, the implication of the precuneus remains yet to be determined. Here we examined the implication of the precuneus in third-person perspective of social interaction using functional magnetic resonance imaging (fMRI). Participants performed a socially irrelevant task while watching the biological motion of two agents acting in either typical (congruent to social conventions) or atypical (incongruent to social conventions) ways. When compared to typical displays, the atypical displays elicited greater activation in the central and posterior bilateral precuneus, and in frontoparietal and occipital regions. Whereas the right precuneus responded with greater activation also to upside down than upright displays, the left precuneus did not. Correlations and effective connectivity analysis added consistent evidence of an interhemispheric asymmetry between the right and left precuneus. These findings suggest that the precuneus reacts to violations of social expectations, and plays a crucial role in third-person perspective of others' interaction even when the social context is unattended

The Bumps and BaBies Longitudinal Study (BaBBLeS): a multi-site cohort study of first-time mothers to evaluate the effectiveness of the Baby Buddy app

Background: Health mobile applications (apps) have become very popular, including apps specifically designed to support women during the ante- and postnatal periods. However, there is currently limited evidence for the effectiveness of such apps at improving pregnancy and parenting outcomes. Aim: to assess the effectiveness of a pregnancy and perinatal app, Baby Buddy, in improving maternal self-efficacy at three months post-delivery. Methods: Participants were 16-years and over, first-time pregnant women, 12-16 weeks gestation, recruited from five English study sites. The Tool of Parenting Self-efficacy (TOPSE) (primary outcome) was used to compare mothers at three months post-delivery who had downloaded the Baby Buddy app compared to those who had not downloaded the app, controlling for confounding factors. Results: 488 participants provided valid data at baseline (12-16 weeks gestation), 296 participants provided valid data at 3 months post-birth, 114 (38.5%) of whom reported that they had used the Baby Buddy app. Baby Buddy app users were more likely to use pregnancy or parenting apps (80.7% vs 69.6%, p=.035), more likely to have been introduced to the app by a healthcare professional (p=.005) and have a lower median score for perceived social support (81 vs 83, p=.034) than non-app users. The Baby Buddy app did not illicit a statistically significant change in TOPSE scores from baseline to 3 months post-birth (adjusted OR 1.12, 95%CI 0.59 to 2.13, p=.730). Finding out about the Baby Buddy app from a healthcare professional appeared to grant no additional benefit to app users compared to all other participants in terms of self-efficacy at three months post-birth (adjusted OR 1.16, 95%CI 0.60 to 2.23, p=.666). There were no statistically significant differences in the TOPSE scores for the in-app data between either the type of user who was engaged with the app and non-app users (adjusted OR 0.69, 95%CI 0.22 to 2.16, p=.519) or those who were highly engaged and non-app users (adjusted OR 0.48, 95%CI 0.14t o 1.68, p=.251). Conclusion: This study is one of few, to date, that has investigated the effectiveness of a pregnancy and early parenthood app. No evidence for the effectiveness of the Baby Buddy app was found. New technologies can enhance traditional healthcare services and empower users to take more control over their healthcare but app effectiveness needs to be assessed. Further work is needed to consider, a) how we can best use this new technology to deliver better health outcomes for health service users and, b) methodological issues of evaluating digital health interventions

Bumps and Babies Longitudinal Study (BABBLES): An independent evaluation of the Baby Buddy app

Introduction: Developments in information and communication technologies have enabled and supported the development and expansion of electronic health in the last decade. This has increased the possibility of self-management and care of health issues.Objectives: To assess the effectiveness of on maternal self-efficacy and mental wellbeing three months post-birth in a sample of mothers recruited during the antenatal period. In addition, to explore when, why and how mothers use the app and consider any benefits the app may offer them in relation to their parenting, health, relationships or communication with their child, friends, family members or health professionals.Design: A mixed methods approach, including a longitudinal cohort study, a qualitative study and detailed analysis and synthesis of data from the Baby Buddy app about the way in which mothers accessed and used the app content.Setting: The study was conducted in five geographically separate sites in England: Coventry, Lewisham, Bradford, Blackpool and Leicester. These areas were chosen as they were geographically, ethnically and socio-economically diverse and where the Baby Buddy app was reported to be well-embedded, both formally and informally, into the maternity and child health pathways by the relevant healthcare staff.Participants: Pregnant women who were aged 16 years and over, had no previous live child, were between 12-16 weeks and six days gestation and booked with the maternity services in each of the five study sites were invited to take part.Interventions: Self-reported use of the Baby Buddy app at one of the three data collection time-points: 12-16+6 weeks gestation, 35 weeks gestation and three months post-birth.Outcome measures: The primary outcome measure was parental self-efficacy at three months post-birth using the Tool to measure Parenting Self-Efficacy (TOPSE). The main secondary outcome was maternal mental well-being at three months post-birth using the Warwick and Edinburgh Mental Wellbeing Scale (WEMWBS).Results: Recruitment took place between September 2016 and February 2017. A total of 488 participants provided valid data at baseline (12-16 weeks gestation), 296 participants also provided valid data at 3 months post-birth, 114 (38.5%) of whom reported that they had used the Baby Buddy app at one or more of the data collection time-points (‘app user’). Seventeen first-time mothers participated in the qualitative arm via telephone interviews (n=9) and a focus group (n=8). Twenty healthcare professionals participated in interviews (n=5) and two focus groups (n=15). Consent was gained from 98 participants who gave permission for their in-app4data to be made accessible but just 61 participants could be identified from the database provided, of whom 51 were included in the analyses.At recruitment there were no differences between Baby Buddy app users and non-app users in respect to: age, IMD, ethnicity, highest education, employment, relationship status. Baby Buddy app users were more likely to use pregnancy or parenting apps (80.7% vs 69.6%, p=.035), more likely to have been introduced to the app by a healthcare professional (p=.005) and have a lower median score for perceived social support (81 vs 83, p=.034) than non-app users. The Baby Buddy app did not illicit a statistically significant change in TOPSE scores from baseline to 3 months post-birth (adjusted OR 1.12, 95%CI 0.59 to 2.13, p=.730). Finding out about the Baby Buddy app from a healthcare professional appeared to grant no additional benefit to app users compared to all other participants in terms of self-efficacy at three months post-birth (adjusted OR 1.16, 95%CI 0.60 to 2.23, p=.666).Apps were popular; Baby Buddy app users were more likely to use other pregnancy-related apps than non-Baby Buddy users and the most frequent source from which Baby Buddy app users found out about the app was a midwife. A post-hoc analysis found that Baby Buddy app users were more likely to breastfeed than non-Baby Buddy app users. This was a consistent pattern for both exclusive breastfeeding and any breast feeding: there was a 9% increase in exclusive breastfeeding at any time up to 3 months post-birth in Baby Buddy app users and a 12% increase in any breastfeeding up to three months post-birth, compared to non-app users. Whilst this is an important finding, this needs to be used with care due to the post-hoc element of the analysis.First-time mothers who participated in the qualitative arm of the study found that the Baby Buddy app worked well due to its accessibility and that the information was concise and easy to find. They liked that it followed the progress of pregnancy with appropriately-timed information and that different aspects could be accessed as and when needed. The app was designed to be an adjunct to service delivery not a replacement for healthcare. The importance of this was demonstrated by many first-time mothers reporting that they preferred in-practice support from a healthcare professional.The qualitative data indicated that the four preconditions of normalisation process theory: implementation, adoption, translation and stabilisation were met in regard to healthcare professionals’ use of the Baby Buddy app. This suggests that the healthcare professionals were actively integrating the Baby Buddy app into clinical practice with other professionals and first-time mothers, therefore embedding the Baby Buddy app into their service delivery.The in-app data from the sub-sample of participants (n=51) suggest that there was a difference in the amount of time participants spent accessing elements of the app; the median time spent using the app per session was 8.3 minutes (SD 5.8 minutes). The most popular features that5were used were ‘Today’s Information’, videos, ‘Bump/Baby Booth’, ‘Ask Me’ and ‘What does that mean?’. Participants used the app most often between 9-10am with another peak in the evening around 8-9pm. There were also a broad range of topics and issues that the participants searched for, of which the most searched words included: ‘labour’, ‘form’, ‘birth’, ‘pregnant’ and ‘developing’. In the sub-sample for whom we had in-app data, there was a large range for the number of times the app was used, from 0-593 times. The median number of times the app was opened was 146.5 but the data were positively skewed (LQ 52.5 – UQ 329). This indicates that the data are bunched towards the smaller number of times opened. Within this sub-sample, 21.6% of the engaged type of user used the app up to 25 times and 47% of this type of user used the app more than 100 times. This contrasts with the highly engaged type of user where 43% used the app 25 or less times and just 9.8% of this proactive type of user used it more than 100 times.We found no statistically significant difference in the TOPSE or the WEMWBS scores between the type of user who was engaged with the app and non-app users (adjusted OR 0.69, 95%CI 0.22 to 2.16, p=.519 and adjusted OR 1.54, 95%CI 0.57 to 4.16, p=.329, respectively). Similarly, we found no statistically significant difference between the type of users who were highly engaged users and non-app users (TOPSE: adjusted OR 0.48, 95%CI 0.14t o 1.68, p=.251; WEMWBS: adjusted OR 1.40, 95%CI 0.52 to 3.76, p=.509).Strengths and limitations: The primary objective was to explore the impact of the Baby Buddy app on parental self-efficacy and the Tool for Parenting Self-Efficacy (TOPSE website, Kendall, Bloomfield and Nash 2009), a validated measure, was selected to measure the primary outcome. The retention rate of 60.7% from baseline to three months post-birth demonstrates the difficulty of engaging new mothers during this demanding period of their lives. Nevertheless, in the initial and final samples, app users and non-users remained generally comparable and relevant confounders were adjusted for. Mothers were invited to take part in interviews and/or focus groups, the latter of which were held in a baby-friendly, welcoming environment for women and babies. Telephone interviews were offered for greater convenience for the women. Analysing the in-app data, we were able to compare outcomes for both the high versus low or non-user app groups and for those mothers who were the type of highly engaged users versus those who were a less engaged type. This was for a relatively small number of mothers but was a new method of analysing the in-app data.The Baby Buddy app was publicly available, meaning randomisation was not possible and therefore participants were only asked about their specific use of the app after the 35 weeks gestation data collection point to avoid directed app use. The participants were a self-selected group, especially those for whom we had in-app data and this is reflected in the higher than the national average for women who were degree holders (58.6% in final sample versus 42% nationally). The overall TOPSE scores were high at baseline which meant there was little room6for improvement. Nevertheless, there was no difference between the Baby Buddy app users and those participants who did not use the app.Conclusions: First-time mothers in the study found the app accessible and the information concise. The quantitative results, including those from the in-app data, found no evidence of impact from the Baby Buddy app on the primary outcome of parental self-efficacy or mental well-being (secondary outcome) at three months post-birth. The participant mothers had lower social support scale scores, which might suggest that the app attracted mothers who had a smaller social support network. Both mothers and healthcare professionals valued the fact that the Baby Buddy app was professionally endorsed which encouraged the women to trust the contents and the healthcare professionals to use it in their everyday practice. The most frequent source from which Baby Buddy app users found out about the app was a midwife, which suggests that the embedding of the app into service delivery by Best Beginnings was beneficial. A post-hoc finding was that women who used the Baby Buddy app were significantly more likely to exclusively breastfeed, or ever breastfeed, than those not using the app. The Baby Buddy app has gone some way to help to ‘Make Every Contact Count’ for both first-time mothers and healthcare professionals